DOCUMENT RESUME 



ED 269 471 



TM 800 328 



AUTHOR 
TITLE 

INSTITUTION 
PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Sanders, Jack; And Others 

A System for Measuring School Effectiveness: The 
Profile of School Excellence (PRO-S/E). 
Appalachia Educational Lab., Charleston, W. Va. 
Apr 86 

8p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (70th, San 
Francisco, CA, April 16-20, 1986). 
Speeches/Conference Papers (150) — Reports - 
Descriptive (141) 

MF01/PC01 Plus Postage. 

Academic Achievement; Elementary Secondary Education; 
^Evaluation Methods; * Identification; * Instructional 
Improvement; Interviews; Profiles; Program 
Descriptions; E ogram Validation; Questionnaires; 
*School Districts; *School Effectiveness; School 
Statistics; Test Results; Validity 
Appalachia Educational Laboratory WV; Profile of a 
School; ^Profile of School Excellence 



ABSTRACT 

This paper reports the status of the Profile of 
School Excellence (PRO-S/E) after three years use and the results of 
a second validation study. The PRO-S/E, used in over 20 school 
districts in six states, is a diagnostic system school districts use 
to determine strengths and weaknesses based on 11 characteristics of 
effective schools. Data is collected by PRO-S/E administrators using: 
(1) district data forms; (2) superintendent interviews; (3) school 
data forms; (4) principal interviews; (5) school rating forms; (6) 
teacher questionnaires; (7) student questionnaires; and (8) optional 
forms to study schools' achievement test scores and students 9 
socioeconomic status. Data from the school rating form and the 
teacher and student questionnaires develop graphic comparison 
profiles for the 11 characteristics. Analyzed results *,re used in 
addition to the interview and descriptive information to develop a 
written comprehensive profile of the school district. The validation 
study resulted in the broadening of the definition of one variable 
category, "Roles and Responsibilities"; the reassignment to other 
variable categories of three items for both the student and teacher 
survey; and the rewarding of five items in the student survey and 
five items in the teacher survey. (PN) 
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Background and Introduction 



As described Ln an earlier paper (Sanders, Shively, and Machesney, 
1982), the Profile of School Excellence (PRO-S/E) is a diagnostic tool 
keyed to 1 instructional and organizational variables research has shown 
to be positively related to effective schools and which are alterable. 
The PRO-S/E operationally defines the 11 variables as follows: 

1. Needs Basis : The extent to which school personnel use an in-place 
system for identifying, teaching, evaluating, and remediating student 
learning needs. 

2. Objectives : The degree to which school personnel prescribe and 
communicate to students relevant and attainable objectives for each 
academic course. 

3. Expectations : The degree to which school personnel communicate 
clearly to each student the belief (the expectation) that each can 
and will succeed in attaining prescribed academic objectives. 

4* Roles & Responsibilitie s: The degree to which school personnel roles 
are defined and understood and the degree to which school personnel 
prepare each student to assume an appropriate level of responsibility 
for learning, to cooperate with others, and to participate in a broad 
range of academic and non-academic activities. 

5» Conditions & Resources : The degree to which school personnel provide 
students exemplary conditions of learning — that is, grouping students 
appropriately; presenting and modeling information and skills in an 
interactive way that properly motivates students; and using excellent 
instructional materials to assure maximum student participation and 
success • 

6. Instructional Time & Task Orientation : The degree to which school 
personnel provide students maximum instructional time during class 
periods and assure that students attend to and successfully engage 
in appropriate academic tasks during class time. 

7. Use of Assessment : The degree to which school personnel use assess- 
ment data as the basis for inforrulng students of their academic 
progress and informing teachers of the^'r students' remediation needs. 

8. Rewards & Reinforcement : The degree to which school personnel use an 
m-place system of reinforcement that recognizes the accomplishments 
and achievements of student and staff. 

9. Code of Behavior : The degree to which school personnel communicate 
clearly and enforce equitably rules, structure, routines, and conse- 
quences governing student behavior. 
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10. School Climate : The degree to which school personnel create and 
model a collegial environment in which students receive and return 
to those around them a sense of caring, personal concern, interest, 
respect, commitment, and support for persons, property, and ideas. 

11. Parental Support & Involvement ; The degree to which school personnel 
have established procedures that encourage meaningful parental and 
community interest, involvement, and support in students 1 academic 
progress. 

The earlier paper defined effective or excellent schools as most 
likely to be found in school districts exhibiting four factors: 

1. Achievement data show that students are learning wnat they are 
supposed to learn and that their socioeconomic status cannot be used 
tc predict their achievement level. 

2. Attendance data show that students and parents value instruction and 
are comfortable with the climate of the schools. 

3. Incidences of delinquency and vandalism are relatively low. 

4. Survey data show that the district gets high grades—high satisfac- 
tion ratings — from students, staff, board members m parents, and 
citizens. 

The purpose of this paper is to report the status of the PRO-S/E 
after three full years of use and a second validation study. The former 
taught that PRO-S/E interviews are more productive if t^e superintendents 
and principals being interviewed fill out the interview protocol indiv- 
idually, before the face-to-face interview. Experience also proved that 
data arrays communicate PRO-S/E reports better than narrative. These 
learnings have resulted in a streamlining of the interview and PRO-S/E 
report preparation process. 

The latter, the validation study, involved researchers sorting the 
items used in the PRO-S/E surveys into the 11 variable categories the 
items purport to measure. The researchers used in this study were faculty 
members at the University of Virginia who are knowledgeable of the school 
effectiveness literature. This study resulted in the broadening of the 
definition of one variable category, "Roles & Responsibilities"; the 
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reassignment to oUier variable categories of 3/74 student survey items; 
the reassignment to other variable categories of 3/82 teacher survey 
items; and the rewording of five items in the student survey and five 
items in the teacher survey. 



Current Status of the PRO-S/E 



The PRO-S/E remains a diagnostic system school districts use to 
determine strengths and weaknesses based on 11 characteristics of effec- 
tive schools* 

Data collection is conducted by PRO-S/E administrators using seven 
required and one optional instruments: 

!• District Data Form : Used to collect data from the superintendent on 
the district's educational philosophy, description of the schools in 
the district, results of any district surveys conducted, district 
budget, rank in state relative to per-pupil expenditure, percent of 
graduates attending postsecondary institutions, standardized test 
results, and community information. 

2* Superintendent Interview : Used to collect data from the superinten- 
dent on his/her professional background, perceptions of the greatest 
local educational needs, programs for improving students' basic 
skills, communication of expectations, encouragement of staff profes- 
sional initiative, evidence of district successes, communication of 
standards /code of conduct, teachers' perceptions of professional 
development, staff roles and goals, staff hiring and orientation 
procedures, monitoring/evaluation of principals, use of research, 
perceptions of school climate, perceptions of community/parental 
involvement, and most serious problems to be faced by the superin- 
tendent and principals during the next five years. 

3» School Data Form : Used to collect data about each school on enroll- 
ment statistics for past three years; average daily attendance; 
pupil/teacher ratio; current innovative programs; any recent survey 
results; student follow-up information; newsletters; policy 
statements on communication of achievement data, use of achievement 
data by teachers and counselors, and school educational philosophy 
including teacher code and student code; and, if a secondary school, 
a listing of courses available. 

4* Principal Interview : Used to collect data from each school principal 
on his/her professional background; school academic goals and commu- 
nication of goals to faculty, students, and parents; perceptions of 
effectiveness of basic skills achievement programs; encouragement 



of staff initiative; use of fiscal resources; attendance policies; 
procedures for monitoring instructional time and academic progress; 
recognition of student academic achievement; rules of classroom 
behavior; staff roles; evaluation of teachers; perceptions of school 
climate; encouragement of parental and community support; and 
problems principals will face in the next five years. 

5. School Rating Form : Used by central office staff to rate each school 
on the 11 PRO-S/E characteristics. 

6. Teacher Questionnaire : A 50 percent sample of noniwinerant teachers 
from each school is selected at random to complete a questionnaire 
that measures perceptions of district and school statu? on the 11 
PRO-S/E characteristics. 

!• Student Questionnaire : A 10 percent sample of students from grades 
6-12 from each school is selected at random to complete a question- 
naire that measures oerceptions of status on the 11 PRO-S/E 
characteristics . 

8. Optional : There is an eighth instrument used to conduct a study of 
the schools 1 achievement test scores and students 1 socioeconomic 
status, Sometimes school districts have such studies on file, 
sometimes not. This study accepts Ron Edmonds 1 finding that in 
effective schools standardized test scores indicate that learning 
opportunities and achievement are similar for students from various 
backgrounds and SES. 

All of the above data collection is conducted by staff of the 
Appalachia Educational Laboratory (AEL) or trained AEL associates. The 
data from the Teacher Questionnaire and the Student Questionnaire are 
collected using mark-sense cards that are read by an electronic reader. 

Data Analysis 

Data from the School Rating Form, the Teacher Questionnaire, and the 
Student Questionnaire are used to develop graphic comparison profiles for 
the 11 characteristics. Analyses of these data and of the optional 
achievement SES substudy data are conducted using a microcomputer and 
software developed by AEL. These results are used in addition to the 
interview and descriptive information to develop a comprehensive profile 
of the school district and the district's schools regarding the 11 



characteristics. 



Report 

A written report is presented to the superintendent that includes the 
following : 

1. Background and purpose of the PRO-S/E. 

2. Results by school on the 11 characteristics. 

3. Results by the 11 characteristics including summary statements for 
elementary schools, summary statements for middle schools, summary 
statements for high schools, ana recommendations. 

4. Overview of the four school effectiveness variables. 

5. Results of the achievement /SES study by school and summarized. 

District Followup 

After the presentation of the written report, PRO-S/E administrators 
contact the district superintendent to answer questions about the results 
and to determine if further services can be used to remediate any of the 
problem areas identified by the PRO-S/E process. 

Users 

The PRO-S/E has been useJ in more than 20 school districts in six 
states, covering a wide range of district sizes. Adopting sites include 
Frankfort Independent Schools (KY), Boyd County Schools (KY), Lockland 
City Schools (OH), Gilmer County Schools (WV) , Pulaski County Schools 
(VA), Knoxville City Schools ( TN) , and Washington Public Schools (PA). 

Future Directions 

School districts continue to erpress interest in the PRO-S/E. As a 
result, the developers have completed and pilot tested a PRO-S/E Training 
Manual which is provided as part of an eight-hour PRO-S/E training work- 
shop to professors and consultants who work closely with local school 



districts and are experienced hands at facilitating R & D-based school 
improvement projects. The workshop has been through both pilot and field 
testing and will be available from AEL by spring 1986. 

PRO-S/E instrumentation will be both expanded and streamlined. 
Expansion will include special adaptations of the current student survey 
so that it may be administered to students in grades one to four — current 
instrumentation is effective with students reading at grade five and 
above. Expansion will include also improvements in the PRO-S/E rating 
form used by district central office staff. Streamlining will include 
printing the surveys on mark-sense score sheets for faster machine 
reading. It will also include use of a more powerful microcomputer than 
the one presently being used. 

Finally, the authors look forward to studying improvement trends 
among school districts which implement PRO-S/E studies. Such studies 
should consider questions about the kinds of changes PRO-S/E districts 
make — are they structured or cosmetic? Are urban school efforts similar 
from district to district? Are they different substantively from those 
of schools in small/rural districts? Do PRO-S/E districts that implement 
improvement plans show improvement on a second PRO-S/E study? What are 
the different ways that districts and principals use PRO-S/E report data 
to improve schools? 



Reference: School Effectiveness: Profile of School Excellence , by Jark 
Sanders, Joe E. Shively, and J. Douglas Machesney. A paper presented at 
the annual meeting of the American Educational Research Association, New 
Orleans, Louisiana, April 1984. 
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